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aturi cutes. Prctlers associated with real time performance, appropriatei 

I 

■methods tc progra'” thes“ architectures . reliability issues, operating!', 
systems Questions, etc. still remain to be solved. It is felt that the 
eristenc^^ of an experimental system whereby any of the ( functicnallyl 
representative) .“.nderson and Jensen architecture types might bej 
conveniently built would provide a powerful tool to study some of these 
proble'-s. This report presents the current "esults of the project which 
has been concerned with designing and building such an experimental 
'Microcomputer syste^’. The philosoohy followed was to base all systems on 
general pupcse hardware primitives from which the various systems can he 
brilt. It is critical to identify these general concents so that the 



results of the exper im^ntal worh can be extended to the general case 



vine ~e different hardware and s o f twa re i mo leme n ta t i o ns occur and 



permit considering the never VLSI devices which are becoming 
c orme r c la 1 l.y available out which are not included in the present 
exoerimental system. 






•esent State of Knowledge 



Advances in VLSI technol ogy have been responsible for the 

propagation of digital technioues for implemerting solutions to 
engineering proble'-s which traditionally have been solved by other 
methods. These advar.c'=>s have also made it possible to consider building 
'Multicomputer systems where until Quite recently digital systems 

utilioir.g a single computer (i.e. one processcr in the system) were the 
norm,. These two trends are c '>nver gi ng so as to ca use a growth of 
activity in consiiering the application of systems of multiple computers 
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guide the generation of oneration sys*e~'s and annlicatio- program's which 
can then be matched to the underlying hardware structure. Any design 
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process ir.vol’/es 'ra’cin^ choices anons alternatives. Information.;: 
descrifcins alternative multicomputer structures is recuirei so that - 
those choices can b= made a systematic fashion. 

•I 

The potential scope of the multiple computer architecture space is.,i 
laCfse eroufth that a study of an important subset of the total space will.,; 
rrovide an important c or tr ibu ti on . .^n area of current interest is, I' 

concerned with special purpose architectures which are based uponll: 
mi c r onr oces s ors ard compatible VLSI devices. ?or the intents of this, 
research j^roject a "special purpose" multiple computer architecture is 
one whose characteris tics ar° fixed at iesisrn time while a 'general 

i 

purpose" architecture is one which may be dynamically reconfigured at i 
run tine acccrdins: to the needs of the tas><!s) it is supporting. The :n 
subject of reliability is of sufficient importance that it will be 
considered as a normal, desirable attribute of special purpose systems.. 
If a hardware module failure occurs r econf i gura t i on cf the system will 
be necessary for the purpose of maintaining the system characteristics , 
which were specified at design time. Thus at least this aspect of ' 
dynamic "econf i^urati on is of interest in special purpose systems. The | 
special purpose syste.ms will encompass the spectrum frc" tightly coupled ^ 
multiprocessors to loosely coup! ed systems . 

From a high level architecture viewpoint, a multiple computer system 
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based processing nodes may be characterized as conventional ^zeneral 
purpose digital computers which execute sequential -rograms. The 
attributes of these nodes mav be described by existing notations such as 
the P'^S and IS? descriptive systems [PellTl ,Sell73l . What is not 
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which to found the study of nultiule computer architectures. Thus a 
rationale exists for deriving a taxcncmy for mul*iple computer systems, 
^t present there are a small numher of existing taxonomies [Anler76] 
[Free79] [Xro72l r?ly72’’ [Cren72’> . Of these taxonomies the one presented 
ty Anierson and Jensen apoears to he a conceptually sound starting point 
for identifying HI?s and ISSs. The nodes in the tree identify high level 
architecture types and provide an initial classification of ISSs. From 
this informa tier, a f unction ally complete set of ISSs and associated HI?s 



may he defined. 

It appears that a •'■ariety of multiple computer configurations are 
to*’h technically and economically feasible. However, detailed 
information needs to be developed which defines the total number of 
iis tinguishable architectures, and their operational characteristics 
before design decisions can be mad“ in a systematic manner. 
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Current -esults from Project : Molular Mic roco'nputers 



4r. eTdT.ple cf a modular rricrocomputer system is the (Microprocessor 
Iniepenient Microcomputer (mimic) [Carey'?7a. Var79l , This is hased upon 
a standardized microprocessor bus, and a set of modular hardware 
pri"’itives which are compatible with any of the current veneration of 9 
bit micrcprccessors . Th^se systems are being extended to the 
Tul t i computer case by definirv harivare primitives from which a variety 
cf systems may be realized. A first set of hardware interconnection 
primitives 'HI?) have beer, iesivned and implemented which include '1) a 
shared memory, 2) multichannel parallel and serial links, ^,3) a 
multinort bus ’window, '5) an intelligent channel processor, and (5) 
extensions of the mImIC bus standard to support shared bus systems. 
These HIPs are beirv installed ir a three node Miyic system so as to 
perform experiments cn a variety of i n te r-' onnec t i on subsystems (ISS) . 
The selection of the particular hardware interconnection primitives was 
done by an analysis of a hivh level taxonomy [.i.nder76l . The designs were 



a function of the characteristics of 
available V‘13 1 devices. It is felt that the 
d will TDrOVldd VdlUd'-ld t ^ t s In 

system’s based unon future VLSI devices and 
and interconnections subsystems. 

Where the '^IMIC multicomputer system 



the MIMIC bus a.ni currently 
experience gained with such 
to uni er s t a nd i ng more complex 
more co"’plei process! nv nodes 

is based upon internal 



University of Connecticut iesivns and 9 bit microprocessors, a second 
multiple computer based upon comf^ercial modules, and the 15 bit LSI 11 
mi c r 0 or oce s s 0 r , has also been designed and implemented. The 
1 r. t ere onnec t i c.” subsyste'^ (133) is entirely based upon serial lin.-cs. 
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Although this is a very low pe^fonar-ce envirooTent the particular ISS 
is very fle.xible, perrrits a variety of architectures tc he i.Tiplenentel 
urder prosran control, and is currently heins used to explore problems 
with distributed software systems. 

Current results : Hard'*’ are Intercom e-^t ion Techn ioues and Primitives 



There are a limited number of ways that digital hardware mcdules and 
system's may be in ter cor.rec * ed . This limitation is based unon the nature 
of digital hardware itself. Additional li-’itations Occur if only certain 
types of digital s ys t e-s are considered sue’’’ as those which are based 
upon microprocessors and associated VLSI devices. In addition to 
hardware li-its fur.c*ional aspects must be considered. In the PMS system 
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d.r ct^cr. To s^y o'“d0 is dr* 

intdrconndction '^dchdo.is'^ rdo'oirds firso sodOir^ Ohs Is 7 si of viswooino 
which is being us^d . ?^r this paper at least thre° viewpoint levels are 
employed . These levels are the '1'' electronic hardware, '2' system 
functional, and (3) so'^tware functional. 
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throush storage in a shared "lertory 'I/O devices rriay he viewed as c ■- 
shared rremory). These ar» the basic possibilities with hardware o 
-rechan s im s of higher functionality bein^ constructed from this set of • 
techniaues . ^ 

At the system functional level the P^IS concepts appear to be *: 
complete. Therefore a given approach will be characterized as a PMS , 
pri'^itive. An interconnection ’"erha”ism may have a completely f 
or osranma b le computer within it cut if the function it is performing is 
that of a switch then a switch it is. ■. 

At the software functional level the activities that the hardware' 
syste-^ is supoortins must be considered. ^cPayden [Mcfay76l suggests 
such a fancticnal viewpoint by proposing the "functional layering" 
concent for organizing software systems in multicomputer systems. In. 
this approach the functional responsibilities of the network nodes are 
divided into three major layers which are (1) transmission management, 
‘2) node functional task -^a nagemen t , and (3) application operations. 
This identifies at least three node tasks which will have a time ordered^ 
relationship with resulting implications about the potential parallelism 
■concurrency) of software operations associated with the node. The 
performance of the node can be immediately affected by the hardware j 
resources in the node which are available to support (exploit) this | 

I 

potential parallelism. 

A further classif ication of the i n t ere on nec t i on mechanisms can be ■! 
made if the system is divided into the two broad classes of internodal 
and intrarodal connections with internodal beins those connections j 
between disjoint nodes and intranodal being those connections within a ,j 
distinct node in the sys*’em. At first it mixht be tempting to envision i; 
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that fundamentally different hardware 
realize in ter nodal versus intranodal conne 
true. As staled th^re are a severely 1 
hardware possibilities for imc lem^r. t in^ ha 
cover both the inter and intra nodal ca 
nay be employed to realize both. The dist 
functional defir. lion as is true with the ? 



mechanisms are necessary 
ctions. However this is 
i mi ted number of basic dic^i 
rdware connections and th 
S“S . The same bard ware trcdu 
iction becomes a matter 
primitives. 



V i t h 

hardware 



the abov= discussion as 
interconnection ori'^itiv=‘5 



ar explanation let the follow 
be defined for mi croccmp iter ca 



systems. 

denerai microcomiputer hardware interconnecti on primitives: 

1 . I^i c rocompu te r bus ' ^ 3 ) : pr a r y i ” t r an o i a 1 c o : ne c t i or. between 
the microprocessor ani all other functional modules in the node. 

2 . c r 00 oppu te r bus window : electronic -appin^ between I'Bs . 

3. Shared *^emory : a me^'ory which ”ay be accessed by mere than 

one microcomputer or "'ioroprocessor. 

4. Shared ^icrooomputer Bus (3*^3' : an internal bus -^hich may be 

dyna-^ically ti~e shared '■y two or --'-e crcprocessors . 



5. Shared Ccm’'uni ca t i on Bus := 'SCB- • = 



, n external bus w’ 



IS sn a 



by two or ~ore *^1 croc o'^ outers for internoial commu-’i cation 
6. Serial Linl-c 'Is) : a bit serial interface whose operations are 
controlled by a "i oroprocessc " . Mr-r’^allv only :sed for trans'~issio 
of informa ti ■T'n . 

7. Parallel Li'k '1?' : a word parallel interface whose Cyeraticns ar 
controlled by a mtcroorocessor . '^ay be used for simole "message 
trans’^i ss 1 on or as part of a control '^®chanism, 

P.Tran smission Processor c?t) : a "icrop''ooessor waose purpose is to 



to 
not 
tal 
e se 
le s 
of 

i nc^ 
sed 



red 
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support the transir-ission layer within a node or between nodes. A 
functional division into two senarate types of ?t may be necessary 
if internodal and intra nodal transmission operations prove to be 
fundamentally different. 

9 . DMA controller jKdma) : implement dm a intranodal transfers. 



There are other possibilities such as an al i^nmen t ne twor< but upon 
eTaT’ir.ation these seem to b® constructed from the above ^^rimitives. 

'liven these basic hardware technicues the next level is to identify 



ha rdva re 
hardware 
taxonomy 
in t h e 
but this 
hardware 
r r. i e r s 0 n 
is that 
yI^^IC h 
1 .Four 



int ercor.’’ “ct i cn primitives which may be used to realize any, 
architecture of interest. In this study the Anderson and Jensen’ 

f 

was employed ♦:o define su^h an architect ire space. As mentioned: 
introductory discussion o*her architectures seem to be possible 



taxonomy seems to 




ti V® 


Therefore the 


basic set of 


technioues were us 


a ^ 


to define FI?s 


from which 


any of the 



and Jensen architectures could be realized. What is of interest 
only a small set of HI ?s is reouired. 
ardware in terconnec tier pri~’itlves : 
c ha n n e 1 serial 1 i n A ( S ; L s J 4 ) : dual 



These are as follows. 



mode to support internodal 



c r 

intranodal information transfers. 

2. Four channel parallel link fS;ip;4'' 
or intrancdal information transfers. 

3. Four channel bus window ■'S;BW;4'' 
support a variety of bus window 

4 . Transm i s s i on computer 'Ctidma) : 

and a Fdma to the bus =r.d 



I 

dual mode to support internodal 

: address mapping is prosrrammabl e to 
based architectures, 
a seoarate computer with a Ip 
which may control either the 



C/) 
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S;Ls;4 or the S;lp:4. This will support either internodal or 
intranodal scftvcr-= " t rar.s.T is s 1 l-=yers." Ir. tr.is HI? tne CtJd-ra 
will support either t ransni ss ior through a Ji'LsIl, a S:lpi4, or 
it provides a irra hi och; t ra r.s f “ r rapa hi 1 i t y within a node. Cost 
modularity c on s ide ra t i ons indicated the efficiency of sroupin^ these 
functions . 

Zach of these modules has been designed and constructed for use in 
th“ systems. Si^ce the system is ■'epresentati ve of systems 



such as the S-10t'’ based micrcco'^puters *Zlm?9] the results of 
imple'^enta ti cn are applicable to S-l?f type systevs disc. 



-nib 



Current Results : •'^ul tic omouter .Architectures 



The following figures ■cres<=nt the oarticular i~?le"en t a t i on s of t.ne 
'^I'^IC HI?s defined above, so'^e of the i n te rco nnec 1 1 on subsystems 






OS si Die vlth oh^se HI?s, er. i 



i w 



: en : ^ : i ^ i c r o c o u t e r 



channel link HI?. This iia^re”' represer'ns the 3 » 1.^*4 sni the 

;Ls^ 4. Ji^-re 4':' anl 4 c ' shov a iia^rran version f these -IPs. 

The pys representation is *^uch clearer ari co'^cact as a ^’^apnical 
notation. An interesting point is that altnc^j^h the functional 
renresertations cf the the 3<lo*4 and the 3»Ls»4 '^'^h'lles are the sa'^e 



their effects on the operational character: sties o 
significantly d i f f e ren t . s an ema-pl = -hys 



system dre 



transmission speeds are ouite different and the S:Lo:4 may also serve in 
a var i e ty cf co r t r c 1 c pe rations which are impossible to the S ; Is 1 4 . Thi s 
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IS a vivid, illustra t,ion cf the need to examine the implemen ta ti or 
considerations b“ycni the functional d<=fini ti c r.s to see the important!’ 

I 

effects of usiri? different interconnection mec’^anisms. ^ 

Jigure 5 presents a ?MS diagram of a typical microcomputer such as|:| 



a or S-130 based machine. ?i;?ure 5 presents a ?i^S dia^jram cf the 
transmission processor HI? vith the various module features defined.; 
Note that this is auite similar in nature to the machine of Figure 5 hut 
the buffered bus connections and the Lp and the Kdma jrovide the 
hardware means to realise the transmission functions. 

Fisure 7 combines the modules of figures 4,5 aid 6 to show a typical 
node which consists of a node nicrocomruter and the HIPs necessary to 






realize a v a r i e i y 



int°rconn®ction coeraticns. This is a general 



purpose structure which will permit implementing a number cf ths 
architectures from the .Inlerson Jensen taxonomy. 
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c h 8 r. 0 1 s i r. 8 


^ i n 0 


h 8 r d- 


ware module 
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cost module ri 


V .7 8 s i 
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reascr.abl e 


to multi 


■ole X 



tne 5* control and ti~in? functions a’^onc separate bus wini'w channels. 

Fieure 9 presents two possible architectures feasible with the 3^14 
HIP. In figure 9 a) is shown a three node conf iguration which permits 
either a DDL type machine or a true IDS configuration. The address ^aps | 
in the bus window are programmac le which permits defining a -apping of 
bus window address spaces which fur. ct i ona 1 1/ define different 
— ^ ^ ''8, vs lo th .0 rin^ 

messages through a store and forward mechanism if tne BV, address maps 



only ■oermi*'ted unidirec*’ional in'^orma*ion t 



n another case th’ 
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3W air°ss raps ray ce set to per it any C in the network tc access any 
other C indirectly through the B*s. In these cases the functional 
soft ware operations and system behavior will differ. 

?isur“ 1? presents a four node "in^ implementation of the structure 
ori 2 inally shown in figure 2^b). In this case the conf igura tion of 
figure 7. which employed the Ct and S:Ls54: HI?s, is used to build a 
higher perfcrr^ance DDL architec'ur®. Other basic structures are also 
possible from this fundamental node. 

Present studies show that nir“ of the ten possible Anderson and 
Jensen architectures are o'^ssible frtm th^se hardware orimitives. Daly 

■^echani s^s . 

?hese systems reflect the possible m-ji : icomouter hardware structures 
a fundamental S“t of building blocks. The next phase of 

is tics which are of 
' ” e X a ~ i n e d in light of a 
^articular HIP/ISS impl e-'en ta t i o n to draw conclusions about the effects 
particular hardware techniques upon system level behavior, prom this 

■; e 1 0 p 1 n g - e : h o i s for 
V “ d s of a given 1 - s i g n 
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tetveen the structure of a iistrihuted software system and the 
underlying; distributed hardware structure. To study this and to permit 
focusing upon the distributed software issues an eight node LSIll 
distributed microcomputer has been designed and is being constructed. 
This system has a general purpose ISS whose structure may be varied 
under program control. The ISS is constructed from serial link HI?s to, 
provide full point to point communication between any node in the system:. 



or 


a r y 


permutation of 


i n t e re s t . 


Th e s c f t va r e 
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def in:=5 
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ticular ISS structure 
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Some 


examples of this 


system are 


shown in figure 


s 10 thr 


ou^h 


12. 


The 



basic system is shown in figure 10 (b) which consists of eight LSIll 



nodes, as defined by figure 10 (a), a full point to point serial lin 



. I 



ISS, and a ?L? 11/50 for program development support. An example of a j 
full point to noint 133 is shown in figure il [c) . Any node can directly';l 
c c^mu.n i ca t e with any other node through d®iicatei serial links. This is 
a cO'-Toletely general sche'~e with some of the possible oermutaticns shown, 
in figures 11 a), ll'b'), ll(d'), ll'e'', and 12. The software system J 

d^firirs d8dic3.t9d 58ridl which it chooses to us0 8.n.d thdt 

defines the ISS. As can be seen this uermits DDL, ICDS, ICDL, DSC, IDDR, 



and IDDI architectures which are six of the ter. basic Anderson and i' 



Jensen structures. This will nrovide a great flexibility to support 
experimental work with distributed software. 

To show other features of this system figure 12 presents a dual DDL 
syste"^. -asically this is a syste.m with an alternate path to compensate 
for a failure of a given dedicated link or the node which suooorts that 



1? 



1 i n K . Such a system provides a r. ercper i'^an t a 1 v e ni c 1 e for considering 



reliability ard software recovery issues. In this systeri two 



000 5 



are physically ief ir.®d . during r. c rral '' pera t i ons only an active m path 
will he employed. In case of a link failure an alternative link will he 
activated. In case of a node failure the node tasks need to he 
tranferred to an active node for a degraded node of operation and 
associated alternative links will h=C'v= active. 

Additional syste"^ a rchi t ec * ur es are cossihle from this general 
purpose serial lir.e I3S. 



Ccn elusions 



It seers t '' be possible to ilertify a set of a rcrii t ec ture types whic: 
■^ay be used to -'ha rac ^eri oe and study multiple <-oruuter systems. It also 
seems possible to identify the fundamental hardware technioues which ar 
employed in i ~ p 1 e m e r. t i n s h a r d wa r “ i r. t e r n o r. r e c t i o r. mechanisms. These two' 
results have been combined in this project to identify an initial set of 






hardware interconnection ■orimitiv=’S (Hips'! which may be 



. 0 bull d* 



functi-'nal representative interconnection subsystems ' ISSs) which resu 



rll 



i i 5 t i neu i sh abl e a re’” i t ec t u-e t yre . An initial set of these HI Ps* 



ha”e been designed and constructed to permit the convenient construction 



^ f thr Vcirious "’'’’Itiols '^icrocoi'i 






r v'n i te c tu re 3 of interest. 



There are a number of -ajor issues which re-^air to be studied prior 
to the develonme-'- of well ordered iesi’n orocedures corcernin=; mrltiole 



mcrcccmputer systems. Can the type; 



1 oossible arch: tec 'U res be 



•• i . . b 0 .. a ij 0. [I 



Jensen tax:ncmv indicates that thus is oossible but a 



identified and cha rac te r i ce d in a meaningful way? T’r.e 

re c 0 m p 1 e 0 e ! 

t a X 0 n 0 ^ y is still n e c e s s e " y w h i ” n will also consider i p 1 e m e ” t a t i o n 
attritutes. Can an enc^moessinm set cf functional, modular orinitives be 
ile''tl:iei w’rich suorort t’re i~'uleme'' o at i on of any m-;ioipie computer 

syste"’? This orcject has acco'^ol Is’ned this f=tcal within a limited 

coriext. This successful result iriica'es that sue’' a- objective "’ey be 
-enerally possible. The vv.5 o" 07 iies a conceptual base for 



his can be done for m’.’l*iole comD'iters. Tees an acoroonate descriotion 



la's’idee e-fi;* whi c'n can be used *o describe, sim’.:lcte, and ~a”ioulate 
tne a ren 1 1 e c tu r e s of oiole computer system 5 ^ suooort for t’.ie 
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[Amd c4 , 


Brovn76] . 


^'1 hat 



is the proper way to characte-ize the rcltiple computer architect 



.- 3 ? 



Perhaps the test way is to cha rac teri z e such systems in terms of their 
^'interconnection su'bsyste"’s which must eventually encompass "both the 
hardware suhsyste'^ and the software suh sy s t . 'jihat are the important 
attributes of this overall int = rccn"ecti on sibsyste'-? Vhat are the 



tradeoffs between hardware and software mechanisms? ’.v'hat is the 
relationshio between the distributed hardwa''e system 3 -^ z'zb distributed 
software system? These and other such Questions remain to studied ani to 



uri*^i*iv05 drid th? 6xistw’ic9 of sx D 0 ri*^ 0 ^. tdl l^scri 

this report will c r 0 v i d e an e f f e ~ i v e e x c e r i m e '' t a 1 means to examine 5 0 ' 
of lh0S0 iSSU8S. 
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^'igure 1 (a) An example of a high speed 
Interconnection subsystem using 
bus windows and a shared memory 



Figure 1(b) An example of a lew 
interconnection subsystem using- 
serial links. 



System characteristics 
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a) DDL : Direct 
Dedicated Loop 



b) DDC : Direct c) DSM : Direct 

Dedicated Complete Shared i-Ieraory 




d) DSB : Direct 
Shared Bus 




e) J.CDS ; _ndirect 
Centralized Dedicated Star 



S 




f) ICDL : Indirect 
Centralized Dedicated loop 




g) ICSE : Indirect 
Centralized Shared Bus 
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*ure ^ (a) 



Block diagram of two mode, four channel link, hardware 
interconnec tion primitive. Above diagram describes both serial 
and parallel link HI?s . 
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Figure ^ {'o] 
TWO mode 



?MS diagram of 
four channel link 






Vi 1 r'c ll / 'l 

owe mode 



Reduced PMS diagram 
four c'^^.a^el link HI? 
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4 charjael bus window HI? 
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Figure 9(a) Three node system with 
Bus Window KI? implementing I3S. 
Possible architectures are DDL if 
window addresses are set sc must do 
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